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ij.  ABSTRACT  (Maximum  200 words) 

The  aims  of  this  research  are  (i)  To  study  the  multidimensional  spatio-temporo-chromatic  signal  processing 
capabilities  of  visual  receptive  fields,  (ii)  To  identify  the  key  attributes  of  multidimensional  image  signals  vi/hich 
are  sensitive  to  receptive  field  filters  and  how  these  attributes  are  transformed  for  encoding  beyond  the 
receptive  fields. 

By  using  digitized  sequences  of  natural  images,  formulating  a  representation  for  natural  images,  and 
incorporating  properties  of  specialized  visual  receptive  fields  and  neural  pathways  we  (i)  Propose  a  m^el  of 
how  the  early  visual  system  efficiently  codes  natural  time  varying  images,  first  by  tracking  part  of  the  image, 
then  by  matching  the  spatiotemporal  properties  of  the  neural  pathway  to  those  of  the  tracked  image.  We  also 
propose  that  retinal  architecture  which  varies  with  eccentricity  also  matches  the  properties  of  the  tracked 
image,  (ii)  Develop  a  new  model  for  signal  propagation  through  multiple  cell  layers  in  the  retirra  which  can 
incorporate  the  different  cell  densities  in  retinal  cell  layers,  (iii)  Propose  that  visual  color  receptive  fields  are 
matched  to  the  spatio-chromatic  structure  of  natural  color  images. 
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1 .  STATUS  OF  THE  RESEARCH 

The  aims  of  this  research  shortly  recapitulated  are: 

(i)  To  study  the  multidimensional  spatio-temporo-chromatic  signal  processing 
capabilities  of  visual  receptive  fields 

(ii)  To  identify  the  key  attributes  of  multidimensional  image  signals  which  are  sensitive 
to  receptive  field  filters  and  how  these  attributes  are  transformed  for  encoding 
beyond  the  receptive  fields. 

Cited  below  are  abstracts  of  papers  published,  in  press,  or  submitted  which 
summarize  the  research.  Full  citation  locations  are  given  in  Section  2.  The 
abstracts  are  given  in  related  groups  and  can  be  disseminated  as  such. 


A.  Spatiotemporal  Coding 

A.1 .  Efficient  Coding  of  Natural  Time  Varying  Images  in  the  Early  Visual  System 

We  investigate  the  hypothesis  that  the  early  visual  system  efficiently  codes  natural 
time  varying  images,  first  by  tracking  part  of  the  image,  then  by  matching  the 
spatiotemporal  properties  of  the  neural  pathway  to  those  of  the  tracked  image.  A 
representation  for  the  time  varying  image  is  formulated  which  consists  of  two 
spatiotemporal  components,  a  velocity  field  component  and  a  stationary  component. 
We  show,  using  digitized  sequences  of  natural  images,  that  the  spatiotemporal 
spectrum  and  other  attributes  of  the  Image  markedly  differ  before  and  after  tracking. 
The  temporal  frequency  bandwidth  and  velocity  distribution  of  the  velocity  filed 
component  are  diminished  in  the  region  of  tracking  and  broaden  with  increasing 
eccentricity  from  this  region.  On  the  other  hand,  the  spectrum  of  the  stationary 
component  is  unaffected  by  tracking.  Comparison  of  the  properties  of  the  tracked 
image  to  those  of  the  M  and  P  pathways  in  the  visual  system  suggests  that  each 
pathway  transmits  different  attributes  of  the  tracked  image.  A  retinal  architecture 
which  varies  with  eccentricity  also  matches  the  properties  of  the  tracked  image. 

A.2.  Separability  of  Spatiotemporal  Spectra  of  Image  Sequences 

We  calculated  the  spatiotemporal  power  spectrum  of  1 4  image  sequences  in  order 
to  determine  the  degree  to  which  the  spectra  are  separable  in  space  and  time,  and 
to  assess  the  validity  of  the  commonly  used  exponential  correlation  model  found  in 
the  literature.  We  expand  the  spectrum  by  a  Singular  Value  Decomposition  into  a 
sum  of  separable  terms  and  define  an  index  of  spatiotemporal  separability  as  the 
fraction  of  the  signal  energy  that  can  be  represented  by  the  first  (largest)  separable 
term.  All  spectra  were  found  to  be  highly  separable  with  an  index  of  separability 
above  0.98.  The  power  spectra  of  the  sequences  were  well  fit  by  a  separable  model 
of  the  form: 
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where  k  is  radial  spatial  frequency,  f  is  temporal  frequency,  and  a,  b  are  spatial  and 
temporal  model  parameters  which  determine  the  effective  spatiotemporal  bandwidth 
of  the  signal.  This  power  spectrum  model  corresponds  to  a  product  of  exponential 
autocorrelation  function  separable  in  space  and  time. 


A.3.  The  Effect  of  Tracking  Strategies  on  the  Velocity  Structure  of  Image 
Sequences 


We  investigate  the  effect  of  different  tracking  strategies,  such  as  local  and  full  field 
tracking,  on  the  mean  and  variance  of  the  image  velocity  field.  We  show  that  while 
local  tracking  reduces  the  velocity  variability  in  an  eccentricity  dependent  manner, 
full  field  tracking  reduces  velocity  variability  equally  across  the  image.  We  test  our 
prediction  with  digitized  image  sequences. 

B.  Retinal  Signal  Sampling 

B.1 .  Signal  Sampling  and  Propagation  through  Multiple  Cell  Layers  in  the  Retina: 

Modeling  and  Analysis  using  Multirate  Filtering 

The  retina  is  a  multilayered  structure.  Each  layer  consists  of  one  or  more  classes  of 
cells,  each  at  its  own  density  and  with  its  own  anatomic  and  physiologic  properties. 
Signals  converge  from  many  cells  in  one  layer  onto  single  cells  in  another  layer,  and 
a  signal  from  a  single  cell  diverges  to  many  cells  in  the  next  layer.  In  this  methods 
paper,  we  develop  a  general  approach  to  retinal  analysis  and  modeling  that 
incorporates  multiple  cell  classes,  their  derisities,  and  related  anatomic  properties. 

The  method  is  based  on  multirate  filtering,  a  branch  of  signal  processing  that 
concerns  manipulating  signals  of  different  sampling  rates.  By  drawing  a 
correspondence  between  cell  density  and  signal  sampling  rate,  we  incorporate 
different  cell  densities,  convergence,  divergence,  variation  in  dendritic  field  shape, 
cell-to-cell  variation  in  synaptic  weights,  and  other  anatomic  features  into  multirate 
models.  We  develop  the  multirate  approach  and  apply  it  to  the  cat  cone  =>  cone 

bipolar  CBbi  =»  on-l3  ganglion  cell  pathway  as  an  example.  We  calculate  the  spatial 
frequency  responses  of  the  CBbi  and  on-f3  cells  based  on  the  cone  spatial  frequency 
response  and  find  that  the  attenuation  of  high  frequencies  in  the  cones  prevents 
aliasing  that  would  otherwise  occur  in  CBbi  and  on-B  cells.  We  compare  the 
calculations  with  cat  psychophysics.  We  show  that  the  optics  of  cat  eye  are 
insufficient  by  themselves  to  prevent  aliasing  in  these  cells,  additional  attenuation  by 
the  cone-cone  gap  junctions  and  cone  aperture  are  necessary.  We  demonstrate 

that  the  highest  spatial  frequency  that  can  be  passed  by  the  retina  without  aliasing  is _ 

not  always  determined  by  the  densities  of  cones,  bipolar  and  ganglion  cells,  but  also  'w 
by  the  synaptic  and  dendritic  weighting  between  these  cells. 


B.2.  Conversions  Between  Parallel  and  Hierarchic  Architecture  Analysis 
Multirate  Filter  Banks 

We  derive  general  conversions  between  equivalent  parallel  and  hierarchic  analysis 
multirate  filter  banks  (MRB's)  as  well  as  sufficient  conditions  for  existence  and 
uniqueness  of  the  conversions.  We  use  MRB's  with  arbitrary,  rational  number 
changes  in  sampling  rate  between  successive  outputs  and  arbitrary  Linear  Time 
Invariant  filtering  for  each  output.  Conversion  consists  of  commuting  sampling  rate 
expanders,  sampling  rate  compressors,  and  filters  to  turn  on  MRB  into  the  form  of 
the  other.  For  a  class  of  MRB's  we  call  "well-formed",  the  conversions  between 
architectures  are  one  to  one. 

B.3.  Complexity  and  Filter  Memory  Requirements  in  Scaled  Gaussian 
Hierarchic  and  Parallel  Analysis  Multirate  Filter  Banks. 

Scaled  Gaussian  analysis  multirate  filter  banks  (MRBs)  are  analysis  MRBs  whose 
filters  are  Gaussians  scaled  differently  in  width  and  height.  They  are  frequently 
employed  in  vision  research  and  image  processing.  We  define  generalized  scaled 
Gaussian  analysis  MRBs  for  both  parallel  and  hierarchic  architectures  and  derive 
conversions  between  them.  We  calculate  the  number  of  multiplications,  number  of 
additions,  and  total  number  of  filter  coefficients  for  both  architectures  implemented  in 
direct  form  and  time-varying  filter  implementations.  Our  results  are  for  both  one¬ 
dimensional  MRBs  and  two-dimensional  MRBs  with  separate  filters.  In  all  cases,  as 
has  been  shown  for  particular  MRBs,  the  parallel  MRB  requires  considerably  more 
multiplications,  additions,  and  filter  coefficients  than  the  equivalent  hierarchic  MRB. 
However,  the  relative  differences  are  far  less  severe  in  the  time-varying  case.  We 
also  examine  the  influence  of  the  resampling  factors  on  the  complexity  results.  The 
results  for  two-dimensional  MRBs  with  separable  filters  are  qualitatively  the  same  as 
in  the  one-dimensional  case. 

Soatio  Chromatic  Coding  fSummarvl 

Neurophysiology  provided  color  research  with  a  wealth  of  single  unit  recordings 
indicating  how  color  is  coded  in  the  early  visual  system.  These  show  that  when  color 
is  coded  in  the  visual  system  it  is  encrypted  together  with  spatial  features.  Filters 
(receptive  fields)  in  the  visual  system  have  spatial  responses  with  varying  spatial 
frequencies  which  are  modulated  by  different  color  (wavelength)  profiles.  Some 
combinations  occur  frequently  in  the  visual  system  and  some  are  rare  or  non¬ 
existent.  An  obvious  requirement  of  the  coding  system  is  that  it  be  sufficient  to 
contain  all  image  information  relevant  to  the  visual  system  for  processing  at  higher 
levels  of  the  visual  system.  One  of  the  most  commonly  applied  hypotheses  to 
understand  the  coding  strategy  is  that  the  visual  system  is  an  efficient  coder  and  that 
it  is  matched  to  its  signal  environment  which  is  natural  color  imagery.  By 
decomposing  natural  imagery  to  its  building  blocks  we  can  compare  how  these  are 
matched  by  the  building  blocks  of  the  visual  image  coding  scheme. 

The  principal  components  (eigenfunctions)  of  a  group  of  signals  can  serve  as  a  basis 
for  that  group.  The  advantages  of  representing  a  signal  by  principal  components 
can  be  briefly  summarized  as  follows,  (i)  The  basis  functions  of  the  representation 
are  orthogonal  and  have  uncorrelated  coefficients,  (ii)  The  expected  square 
coefficients  (eigenvalues)  are  minimum  compared  to  other  generalized  Fourier 
representations.  These  properties  make  principal  components  attractive  for  efficient 


signal  coding  and  representation,  because  uncorrelatedness  reduces  redundancy 
and  smaller  coefficients  require  smaller  dynamic  ranges  and  hence,  less 
transmission  resources.  Similar  considerations  have  been  used  before  in  identifying 
the  strategy  of  visual  system  operations.  The  eigenfunctions  of  black  and  white 
images  are  well-known.  They  include  functions  with  periodic  profiles  in  both  spatial 
dimensions.  We  extended  the  analysis  of  the  one-plane  black  and  white  images  to 
3-plane  color  images. 

We  computed  the  principal  components  of  a  number  of  natural  color  images  using 
small  sections  of  the  images.  We  used  3-D  blocks  of  4  x  4  pixels  in  each  color 
plane.  The  visual  system  also  applies  receptive  fields  which  cover  small  sections  of 
the  image.  Principal  components  computed  for  the  natural  color  images  are 
comprised  of  basis  functions  with  various  spatial  and  color  profiles.  These  principal 
components  are  similar  to  certain  visual  system  image  operations  in  space  and  in 
color,  including  the  color  opponent  transformation  and  a  variety  of  spatially 
organized  receptive  field  filters  with  different  orientations.  Color  opponent  profiles 
are  most  prominent  in  low  spatial  frequencies  and  have  very  low  eigenvalues  for 
high  spatial  frequencies.  For  example,  the  fraction  of  signal  energy  represented  in 
low  spatial  frequency  color  profiles  is  significant,  while  the  expected  signal  energy  in 
color  profiles  with  high  spatial  frequency  is  negligible.  This  suggests  that  color  and 
space  are  nearly  separable  when  small  sections  of  natural  images  are  considered. 
This  separability  diminishes  as  the  size  of  the  sections  increase.  Another  advantage 
of  the  small  sections  is  their  computational  load.  The  computational  load  associated 
with  whole  images  or  large  sections  thereof,  in  this  kind  of  analysis,  is  immensely 
large.  We  reconstructed  the  image  using  the  eigenfunctions.  We  find  that  the  image 
can  be  reconstructed  from  a  small  number  of  eigenfunctions  which  are  similar  to 
visual  system  receptive  fields. 
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